Beyond Periodicity: Towards a Unifying Framework for Activations in Coordinate-MLPs

Coordinate-MLPs are emerging as an effective tool for modeling multidimensional continuous signals, overcoming many drawbacks associated with discrete grid-based approximations. However, coordinate-MLPs ReLU activations, in their rudimentary form, demonstrate poor performance representing signals high fidelity, promoting the need positional embedding layers. Recently, Sitzmann et al. [24] proposed a sinusoidal activation function that has capacity to omit from while still preserving signal fidelity. Despite its potential, ReLUs dominating space of coordinate-MLPs; we speculate this is due hyper-sensitivity networks – employ such activations initialization schemes. In paper, attempt broaden current understanding effect coordinate-MLPs, and show there exists broader class suitable encoding signals. We affirm only single example class, propose several non-periodic functions empirically more robust against random initializations than sinusoids. Finally, advocate shift towards these non-traditional simplicity (Code available at https://github.com/samgregoost/Beyond periodicity ).